The Interactive-Music Network DE4.7.2 CIMS: Coding Images of Music Sheets

This document reports the applications and practices in the domain of coding images of music sheets (music imaging), which include music sheet digitisation, recognition, restoration, and others. It reports hardware and software related to music imaging, with discussions on main obstacles and approaches to evaluate state of the art OMR system. Keyword List: Music imaging, music digitisation, sheet music, image processing, scanner, optical music recognition, OMR, optical music restoration, multimedia, image DE4.7.1 — Coding Images of Music MUSICNETWORK Project 2 Table of Content 1 EXECUTIVE SUMMARY AND REPORT SCOPE ........................................................................................3 2 INTRODUCTION.................................................................................................................................................3 3 BACKGROUND ...................................................................................................................................................3 4 OBSTACLES, MUSIC NOTATION ..................................................................................................................4 5 MUSIC DIGITISATION......................................................................................................................................4 5.1 HARDWARE .....................................................................................................................................................4 5.2 DIGITISATION..................................................................................................................................................6 6 OMR .......................................................................................................................................................................7 6.1 COMMERCIAL OMR SYSTEMS........................................................................................................................7 6.2 RESEARCH OMR SYSTEMS.............................................................................................................................7 6.2.1 OMR Granularity.......................................................................................................................................7 6.2.2 On-line and Off-line OMR ........................................................................................................................8 6.2.3 OMR Research and Development .............................................................................................................8 7 OMR EVALUATION.........................................................................................................................................11 7.1 OBSTACLES ...................................................................................................................................................11 7.2 THE OMR QUICK-TEST ................................................................................................................................13 7.3 EVALUATION OF PERFORMANCE BASED ON COMPLETE MUSIC SYMBOLS AND RELATIONSHIPS RECONSTRUCTION.......................................................................................................................................................14 8 MUSIC IMAGE RESTORATION AND PRESERVATION ........................................................................18 8.1 MUSIC IMAGE RESTORATION........................................................................................................................18 8.2 MUSIC PAPER PRESERVATION ......................................................................................................................19 9 APPLICATIONS AND FUTURE DIRECTIONS ..........................................................................................20 10 REFERENCES ...............................................................................................................................................21 11 OMR BIBLIOGRAPHY................................................................................................................................24 DE4.7.1 — Coding Images of Music MUSICNETWORK Project 3 1 Executive Summary and Report Scope This document reports the applications and practices in the domain of coding images of music sheets (music imaging), which include music sheet digitisation, optical music recognition (OMR) and optical music restoration. With a general background of Optical Music Recognition (OMR), the report discusses typical obstacles in this domain and reports currently available commercial OMR software. It reports hardware and software related to music imaging and discusses steps required to evaluate the state of the art OMR system. Besides the main focus on the transformation from images of music scores to symbolic format (for printed and handwritten music notation), this document also reports music image restoration and the application of music imaging techniques for graphical preservation and potential applications for crossmedia integration. 2 Introduction The document explore issues on the digitisation, restoration and automatic transcription of music documents; converting paper-based music document into machine readable formats, in order to explore effective use of the latest interactive and multimedia technologies for cultural heritage restoration and preservation of musical documents, such as printed music scores, handwritten manuscripts and ancient music scores. With the advancements of digitisation and information technologies, document analysis and optical character recognition technologies are now widely used, from form processing to handwritten address recognitions. As we know, document imaging, analysis and understanding is extremely complex, not to mention the additional complexities inherent to Music notation. There are a vast amount of invaluable paper-based heritage, including printed music scores and handwritten manuscripts that are deteriorating over time due to natural decaying of paper and chemical reaction (e.g. printing ink and paper). This situation is a similar challenge to many other paper-based items in library and museum archives. In order to introduce interactive multimedia music capabilities and functionalities, machine readable representation is required. Hence one of the main steps is to create digital version of these paper-based heritage materials for further processing (restoration, encoding, recognition etc) in order to allow long term preservation and wider and more effective distributions. Various efforts have been focused on this issue in order to preserve the record of our heritage. For example, manual and highly skill paper-splitting technique used to conserve Bach’s manuscripts [Porck & Teygeler, 2000; Wächter et al., 1996] and others. 3 Background Digitisation has been commonly used as a possible tool for preservation. Although the digital copy may not conserve the original document, it can preserve the data in the document, with the advantage of easy duplications, distribution and digital processing. Optical Music Recognition (OMR), also commonly known as OCR for Music (Optical Character Recognition for Music) was first attempted in the 60s, and since then there have been a wide range of research and developments in this interdisciplinary domain. Currently there are various commercially available products as well as research systems for OMR. OMR system transforms paper-based printed music scores and handwritten music manuscripts, into a machine-readable symbolic format, and an optical music restoration system to reconstruct small discontinuities and imperfection in the musical writings, including broken stems and stave lines. An idealise system which could reliably “read” and “understand” music notations could provide a wide range of applications for interactive multimedia music, bringing paper-based music to the new multimedia era. DE4.7.1 — Coding Images of Music MUSICNETWORK Project 4 OMR was first attempted over thirty years ago [Pruslin, 1966]. It has received much attention over the last fifteen years [Bainbridge & Wijaya, 1999; Bellini et al., 2001; Bruno & Nesi 2002; Ng & Boyle, 1992; Ng, 1995; Ng et al., 1999; Ng, 2002; etc, see Section “OMR Bibliography”], and there are currently a number of commercially available packages, such as capella-scan [capella-scan], Optical Music easy Reader [OMeR], PhotoScore [PhotoScore], SharpEye [SharpEye], SmartScore [SmartScore] and Vivaldi Scan [Vivaldi Scan]. However there are still much room for improvements in many aspects. Reviews and background on the development of various OMR systems can be found in Bainbridge & Carter [1997], Blostein & Baird [1992] and Selfridge-Field [1994]. An online bibliography on OMR can be found at the Interactive MUSICNETWORK website (http://www.interactiveMUSICNETWORK.org) and http://www.kcng.org/omrbib/ 4 Obstacles, Music Notation Optical Character Recognition (OCR) is perhaps the best known related document image processing problem, but OMR can be critically different. The visual problem might seem simple since writing is normally black on white paper. However, OMR introduces an additional layer of complexity due to the wide range of possible shape variation resulted from inter-connections and groupings of symbols. Furthermore there may be other symbols (e.g. expressive signs, fingerings, bowing, texts, etc.) that are positioned around and sometime overlaid part other music symbols (for example, a tie crossing a stem or touching a note-head). Music Notation is inherently opened ended. Even if generally considered as stable for the period of XVIIIth and XIXth centuries in the Western world, there are several exceptions, such as “unmeasured notation” (for cadenzas and so on), approximate rhythmic notation (several examples can be found in works of authors like Chopin, Schumann or Mendelssohn), or slight enhancements to traditional notation (slurs without ending note, non canonical time signatures...). In the earlier centuries, with neumatic or Gregorian notation, music notation was very far of a standardized system, and in the XXth century, music notation has exploded, and is noticeably far from that model commonly known as Common Western Music Notation. Direct recognition of musical symbols is difficult due to the design of the notation. In general, OMR system uses divide-and-conquer approaches to separate musical features before recognition. For example, stave lines are detected and marked before/after note-head in order to separate one feature from the other. Basic musical syntax (e.g. time-signature) and domain-knowledge enhancement such as rhythmical analysis have been explored to improve recognition performance. Fahmy & Blostein [1998, 1994] propose a graph-rewriting approach for OMR enhancement. Stückelberg et al. [1997] propose an architecture for OMR with high-level domain knowledge and Stückelberg & Doermann [1999] explore probabilistic reasoning for musical score recognition. Coüasnon [2002] comments that existing OMR software is not suitable for industrial context due to time consuming and tedious manual proof reading, and proposes a system that is capable of self-diagnostic to detect error [Coüasnon and Rétif, 1995]. The paper discusses the application of musical knowledge of music writing to enhance OMR processing and recognition using DMOS (Description of MOdification of Segmentation), a generic recognition approach for structured document analysis with grammatical formalism EPF (Enhanced Position Formalism). 5 Music Digitisation 5.1 Hardware Nowadays, document digitisation systems such as optical flatbed scanners are widely available. There are a wide range of commercial products from manufacturers such as Fujitsu, Agfa, HP, Cannon, Epson, UMAX, Microtek, Visioneer and many more. Currently available commercial products are equipped with USB, parallel or SCSI interfaces. Some of these products support dual-interfaces. DE4.7.1 — Coding Images of Music MUSICNETWORK Project 5 Many of these products are capable of more than 600 d.p.i. (dot per inch) optical scan resolution with grey or up to 48-bit colour depth which surplus general requirement for OMR processing. Increasingly digital photo-copiers are also equipped with optical scanner which provides high-speed digitisation. Examples include products from Ricoh and Canon. Drum scanners are less commonly being used in this domain. Besides professional flatbed scanners (such as Creo Scitex, Heidelberg and others), industrial music imaging applications for archiving (bitmap images) also use a digital-camera-back or digital-camera with a copy-stand setup which range from a simply board for document placement to include fully pneumatically controlled book cradle system as well as complex robotic control automatic page-turning system. Examples of overhead-scanning products include: Company Product Notes URL Kirtas Technologies, Inc. (USA) APT BookScan 1200 World's first automatic book scanner http://www.kirtastech.com 4DigitalBooks "DIGITIZING LINE" Automatic digitizing system http://www.4digitalboo ks.com Zeutschel GmbH various MONISCAN models Large format colour scanner OMNISCAN9000 http://www.zeutschel.d e Solar Imaging Systems, UK M3 & M5 digital camera systems Maximum optical resolution 8192x12000 pixels http://www.solarimaging.com Icam Archive Systems, UK GUARDIAN Various models including Guardian which uses Phase One camera backs http://www.icamarchiv e.co.uk Konica Minolta Minolta PS7000 book scanner Up to A2, 256 greyscales http://www.minoltaeuro pe.com/... InfoSys GmbH alpha librae Up to 900 pp/hour, greyscale & colour model http://www.infosysscanner.de/indexE.html ImageWare Components GmbH Bookeye products Oversize formats up to 350 x 720 x 470 mm http://www.bookeye.co m Imaging Business Solutions SMA ScanFox A1 and A2 http://www.imagingbus iness.co.uk Lumiere Technology Jumbo Scan 30000x12000 pixels http://www.jumboscan. com/ Cruse Digital Equipment Various models including Synchron Table Scanners CS 175P which accepts originals as large as 40"x60" http://www.crusedigital .com/scanners.html Zeutschel GmbH Zeutschel Omniscan 10000 Bbooks, newspapers and large format documents (maps, drawings, posters) 871x 610 mm (A1) = 10424x 7300 pixels and 24 bit/pixel http://www.zeutschel.d e Table 1: sample document digitising hardware With increasing pixel count, one-shot digital camera systems are increasingly usable for this domain. For example: • PhaseOne, www.phaseone.com • BetterLight, www.betterlight.com • Imacon, www.imacon.dk • Fujitsu, http://www.fujitsu.com and DE4.7.1 — Coding Images of Music MUSICNETWORK Project 6 • others With high-end digital camera or scan-backs system, copy-stand is necessary. Examples of copy-stand include: • Bencher, http://www.bencher.com/copystands.html • Beseler, http://www.beselerphoto.com/Product_Catalog/o1.pdf • Kaiser, http://www.kaiser-fototechnik.de • Linhof, http://www.linhof.de/english/zubehor/repro/repro.html • Testrite, http://www.testrite.com/CopyStands.htm • Tarsia Technical Industries, http://www.ttind.com The section has presented some sample hardware for the digital acquisition of musical documents. This is a fast moving market and hence it is simply as a guide, with some special mention on relevant hardware such as the book scanner and high quality large format digitiser which are important for music manuscripts preservation, to provide general information. More up-to-date information is to be maintained on the project website. In addition to hardware system, the following sub-section discusses digitisation guidelines and relevant issues for digitising musical documents.